A Database for the Exploration of Spanish Planning

We describe a new task-based corpus in the Spanish language. The corpus consists of videos, transcripts, and annotations of the interaction between a naive speaker and a confederate listener. The speaker instructs the listener to MOVE, ROTATE, or PAINT objects on a computer screen. This resource can be used to study how participants produce instructions in a collaborative goal-oriented scenario, in Spanish. The data set is ideally suited for investigating incremental processes of the production and interpretation of language. We demonstrate here how to use this corpus to explore language-specific differences in utterance planning, for English and Spanish speakers. 1. Task-based Multimodal Corpus in Spanish We present the Spanish Language Fruit Carts corpus based on Aist et al. (2006). This is a video-taped data set of interlocutors instructing a confederate to manipulate objects on a screen. Speakers were free to use any language they chose. The listener uses the mouse to execute the speakers’ request and does not give verbal feedback. Hence, the data set is a multimodal corpus formed by interleaving speakers’ gestures, spoken instructions, and object manipulations with a mouse. In each video, a naive speaker and a confederate listener collaborate in executing a common task. The speakers’ goal is to replicate a given map by instructing the listener on how to MOVE, ROTATE, or PAINT objects on the computer screen (Figure 1). Since the environment on the computer screen and the reference map differ in the objects’ locations, orientations, and colors, the speaker needs to provide elaborate instructions to the listener based on the reference map. The corpus consists of 120 digital videos of 15 Spanish speakers,undergraduate students, recruited from Universidad de Oriente in Valladolid, Mexico, and Harvard University in Cambridge, MA. Each video ranges from 4 to 8 minutes in duration, with an average of 240 utterances per speaker. The speech was transcribed and annotated by two research assistants. 2. Previous Task-Based Corpora Corpora can be used to understand how people interpret and produce language-as-action (Clark, 1992). Towards this end, corpora that capture interactive (human-human or human-machine) communication during the execution of a joint activity plays an important role. Various efforts have addressed the need for this type of resource: ATIS (Dahl et al., 1992), TRAINS (Heeman and Allen, 1995), and Maptask (Anderson et al., 1991). In the ATIS corpus, participants were asked to inquire about air flights reservations, while interacting with a Wizard of Oz (i.e., a human emulating a dialogue system (Kelley, 1985)), or directly with a dialogue system. In the TRAIN Figure 1: Sample map in privileged view to the naive speaker. Speaker and listener both see the current state of affairs on the computer screen. corpus, participants were given a task of transporting oranges to factories, making orange juice, and moving orange juice. One of the participants instructed a second one, who played the role of an assistant in carrying out these tasks. In the Maptask corpus, two participants were each given a map which differed slightly from each other: only one of the maps depicted a route, and it had objects in different locations. The participants’ task was to successfully draw the route on the map that lacked one. In summary, these corpora provide rich information about task-based collaborative interaction. They are also based on mono-lingual data sets collected in English. The corpus differs from previous task-based corpora in three ways: a) the type of task that participants execute, b) the data annotation scheme, and c) availability of comparable corpora in multiple languages. In terms of the task, unlike ATIS, our participants learned the goal of their task in a visual, non-linguistic manner. Thus, the task was not testing memory accuracy or capacity; neither was speakers language directly influenced. Unlike Maptask, our task was richer, in that participants had a variety of well defined actions that could be performed (e.g., MOVE, ROTATE, and PAINT). Unlike TRAINS and Maptask, the in-